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PREFACE 


This  report  presents  an  analysis  of  Government-sponsored  research  and  develop¬ 
ment  efforts  in  the  area  of  intelligent  tutoring  systems  (ITS).  The  work  was  performed  as 
part  of  a  Central  Research  Project,  Information  Technology  for  Education  and  Training.  An 
earlier  report,  IDA  Paper  P-3003,  Government-Sponsored  Research  and  Development 
Efforts  in  the  Area  of  Intelligent  Tutoring  Systems,  contains  the  original  data  and  was  pub¬ 
lished  earlier  to  make  the  data  available  in  a  timely  manner  to  researchers  in  the  field  of  ITS. 
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EXECUTIVE  SUMMARY 


This  report  reviews  the  current  research  and  development  (R&D)  activities  of  Gov¬ 
ernment  organizations  in  the  area  of  intelligent  tutoring  systems  (ITS)  to  provide  informa¬ 
tion  that  can  be  used  by  decision  makers  in  determining  the  composition  of  future  ITS  R&D 
programs. 

In  all,  information  was  collected  about  41  current  and  6  planned  ITS-related 
projects.  These  projects  ranged  from  multi-million  dollar,  multi-year  productization  efforts 
to  small-scale,  six-month  Small  Business  Innovation  Research  Phase  I  investigations  of 
particular  technical  issues. 

ITS  R&D  remains  a  small  but  active  field.  Based  on  the  analysis  of  current  and 
planned  Government-sponsored  R&D  efforts,  10  findings  have  emerged: 

•  The  Department  of  Defense  (DOD)  continues  to  be  the  major  player,  with  the 
Air  Force  Armstrong  Laboratory  taking  the  leading  role,  in  accordance  with  the 
Defense  Science  and  Technology  Reliance  Initiative. 

•  Component  and  architectural  ITS  issues  are  stabilizing  and  there  are  some  indi¬ 
cations  of  a  maturing  technology.  Currently,  popular  areas  of  research  are  stu¬ 
dent  diagnosis,  interactive  dialogue,  and  virtual  reality  applications. 

•  There  is  evidence  of  a  push  towards  ITS  technology  transfer,  with  the  Air  Force 
sponsoring  two  major  efforts  that  correspond  to  roughly  12%  of  the  total  fund¬ 
ing  in  the  ITS  area. 

•  For  ITS  intended  for  practical  use,  the  predominant  application  domain  remains 
troubleshooting  of  electronic  devices.  The  range  of  application  domains  for 
proof-of-concept,  prototype,  and  other  research  ITS  is  growing. 

•  ITS  systems  intended  for  practical  use  do  not  exhibit  high  levels  of  intelligence. 

•  Only  in  about  one-third  of  the  cases  are  ITS  development  efforts  driven  by  pre¬ 
defined  training  requirements:  these  systems  do  not  correspond  one-on-one 
with  systems  intended  for  practical  use. 

•  There  is  much  development  of  ITS  authoring  systems,  taking  roughly  one-third 
of  total  current  spending  in  the  ITS  area. 


♦  More  attention  is  being  paid  to  formally  evaluating  ITS  effectiveness.  Roughly 
one-third  of  ITS  under  development  are  planned  for  summative  evaluation,  but 
critical  evaluation  issues  remain  to  be  addressed. 

«  There  are  some  small  efforts  looking  at  issues  of  cognition  and  pedagogy,  and 
Armstrong  Laboratory  is  in  the  early  stages  of  establishing  a  laboratory  for 
long-term  investigation  of  pedagogical  issues. 

•  Two  groups  are  actively  including  teachers  or  instructors  in  their  efforts.  A  few 
efforts  are  building  capabilities  in  ITS  that  are  designed  to  help  teachers  in  then- 
use  of  the  systems.  Nonetheless,  there  is  a  lack  of  understanding  on  how  teach¬ 
ers  need  to  be  supported. 

In  light  of  these  findings,  the  following  recommendation  are  made: 

Recommendation  1:  More  focus  is  needed  on  evaluation  issues.  The  next  sev¬ 
eral  years  should  see  summative  evaluation  of  all  ITS  developed  for  practical 
use,  while  moving  towards  assessing  the  effectiveness  of  ITS  technology  in 
general. 

Recommendation  2:  The  current  level  of  effort  in  the  area  of  authoring  systems 
should  not  be  expanded  at  the  present  time.  While  authoring  systems  can  be 
valuable  tools  for  researchers,  many  issues  need  to  be  addressed  before  more 
such  tools  are  developed  for  use  by  the  general  educational  community. 

Recommendation  3:  More  effort  is  needed  in  investigating  how  to  prepare  and 
support  teachers  in  their  use  of  ITS  technology. 

Recommendation  4:  The  role  of  ITS  in  supporting  new  instructional  approach¬ 
es,  such  as  learner  control,  collaborative  learning,  and  situated  learning,  should 
be  investigated  in  addition  to  virtual  reality  and  learning. 

Recommendation  5:  Continued  transition  of  the  technology  into  practice  use 
should  be  encouraged.  Public  awareness  of  the  potential  of  this  technology 
needs  to  be  increased. 

The  importance  of  education  and  training  is  increasing,  both  in  Government  and 
industry.  This  is  in  response,  in  part,  to  such  drivers  as  the  increasing  need  for  a  flexible 
workforce  and  DOD  downsizing.  The  data  are  not  available  to  see  how  current  levels  of 
funding  differ  from  spending  in  previous  years,  but  declining  Government  and  DOD  bud¬ 
gets  may  lead  to  substantial  ITS  R&D  funding  reductions  in  future  years.  The  potential  of 
this  slowly  maturing  technology  must  be  realistically  assessed  so  that  appropriate  contin¬ 
ued  R&D  funding  can  be  determined  to  ensure  a  worthwhile  pay-off  in  future  years. 
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1.  INTRODUCTION 


This  report  reviews  the  current  research  and  development  (R&D)  activities  of  Gov¬ 
ernment  organizations  in  the  area  of  intelligent  tutoring  systems  (ITS)  to  provide  informa¬ 
tion  that  can  be  used  by  decision  makers  in  determining  the  composition  of  future  ITS  R&D 
programs.  Specifically,  it  summarizes  the  current  and  planned  activities  of  several  Depart¬ 
ment  of  Defense  (DOD)  organizations,  the  Department  of  Education  (DOEd),  the  National 
Aeronautics  and  Space  Administration  (NASA),  and  the  National  Science  Foundation 
(NSF).  The  particular  Government  organizations  that  provided  information  for  this  report 
are  identified  in  Figure  1 . 
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Figure  1.  Government  Organizations  Providing  Information 

In  all,  information  was  collected  about  41  current  and  6  planned  ITS-related 
projects.  These  projects  ranged  from  multi-million  dollar,  multi-year  productization  efforts 
to  small-scale,  six-month  Small  Business  Innovation  Research  (SBIR)  Phase  I  investiga¬ 
tions  of  particular  technical  issues. 


The  next  chapter  of  this  report  provides  a  quick  introduction  to  ITS.  The  third  chap¬ 
ter  identifies  the  ITS-related  R&D  projects  of  the  participating  organizations,  looking  at 
both  current  and  planned  projects  and  summarizing  the  major  focus  of  each  effort.  The 
fourth  chapter  presents  the  findings  of  the  analysis  based  on  the  issues  identified  above.  The 
report  concludes  with  a  set  of  recommendations  pertaining  to  future  funding  in  the  ITS 
area. 
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2.  BACKGROUND  TO  INTELLIGENT  TUTORING  SYSTEMS 


Individualized  tutoring  has  long  been  recognized  as  offering  a  two-standard  devia¬ 
tion  improvement  in  student  achievement  over  what  is  achieved  by  traditional  classroom 
teaching  methods  [Bloom  1984].  ITS  are  intended  to  make  the  advantages  of  individualized 
tutoring  widely  available  through  the  use  of  modern  computer  technology.  Essentially,  an 
ITS  is  a  computer-aided  system  that  acts  like  a  personal  tutor  in  presenting  instruction  and 
remediation  in  a  manner  tailored  to  each  individual,  based  on  a  gradual  accumulation  of 
knowledge  about  that  individual. 

The  majority  of  early  computer-aided  instruction  systems  were  either  tutorial  or 
drill-and-practice  systems.  In  both  these  forms  of  computer-aided  instruction,  the  student 
is  led  through  the  material  to  be  taught,  providing  a  response  at  every  step.  Based  on  these 
responses,  the  student  is  moved  ahead  to  the  next  part  of  the  material,  or  else  the  current 
part  is  reviewed  until  subsequent  responses  indicate  correct  understanding.  The  actions  tak¬ 
en  by  the  system  to  student  responses  are  mostly  preprogrammed.  Researchers  began  inves¬ 
tigating  how  to  tailor  instructional  actions  to  individual  students  and  this  led  to  the  type  of 
computer-aided  instruction  systems  called  intelligent  tutoring  systems. 

The  definition  of  an  ITS  has  changed  little  since  the  initial  development  of  this  tech¬ 
nology  three  decades  ago.  As  defined  by  Hartley  and  Sleeman  [1973],  ITS  use  three  types 
of  knowledge  to  support  computer-aided  instruction;  that  is,  knowledge  of  the  domain 
(expert  model),  learner  (student  model),  and  teaching  strategies  (tutor).  The  key  character¬ 
istics  that  distinguish  an  ITS  from  other  forms  of  computer-aided  instruction  are  the  follow¬ 
ing  [Fletcher  1988]: 

•  ITS  are  information  system  oriented.  The  underlying  structure  of  the  system  is 
based  on  a  knowledge  base  of  information  about  the  subject  matter,  as  opposed 
to  preprogrammed  blocks  of  static  material  [Carbonell  1970]. 

•  ITS  are  generative.  The  ITS  is  capable  of  generating  instructional  material  from 
the  knowledge  base,  for  example,  to  ask  a  question  designed  to  resolve  a  stu¬ 
dent’s  misunderstanding. 
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•  ITS  provide  for  mixed  initiatives,  that  is,  either  the  system  or  the  student  can 
initiate  an  interaction. 

(As  a  word  of  caution,  the  reader  should  note  that  not  all  the  systems  discussed  in  this 
report  exhibit  these  characteristics.  The  systems  included  here  are  those  put  forward  as 
ITS  by  their  developers,  or  those  commonly  discussed  in  the  literature  as  ITS.) 

In  the  simplest  terms,  an  ITS  works  by  having  a  student  solve  problems.  The  stu¬ 
dent’s  performance  is  monitored  by  reference  to  an  expert’s  behavior,  and  the  system  builds 
and  maintains  a  dynamic  model  of  the  student’s  knowledge  and  skill.  This  model  may  use 
an  overlay  approach  or  employ  misconception  identifiers  to  capture  information  about  stu¬ 
dent  misconceptions.  In  the  first  case,  the  model  just  represents  what  a  student  does  with 
respect  to  what  an  expert  would  do,  treating  the  student’s  misconception  as  a  hole  in  his 
knowledge.  In  the  second  case,  the  ITS  tries  to  identify  the  cause  of  the  misconception.  In 
any  event,  the  ITS  uses  the  student  model,  and  information  on  what  the  student  needs  to 
know,  to  determine  what  to  teach  next  and  how  to  present  it. 

Since  the  early  years,  the  key  research  issues  have  related  to  both  artificial  intelli¬ 
gence  (AI)  technology  and  cognitive  psychology.  For  AI,  the  bulk  of  the  research  focused 
on  what  knowledge  should  be  captured  in  the  different  models,  how  to  represent  that 
knowledge,  and  how  to  diagnose  and  respond  to  student  errors.  The  technology  character¬ 
istics  of  some  early  military  ITS  have  been  reported  Fletcher  [1988]  and  are  summarized 
in  Table  1  to  give  a  feel  for  the  type  of  technology  used  in  the  1980s.  The  typical  application 
areas  for  these  systems  were  maintenance  and  operator  training. 

The  main  thrust  from  the  cognitive  psychology  side  has  been  looking  at  how  to 
involve  a  student  more  actively  in  the  learning  process  and  reflecting  changes  in  popular 
pedagogy  in  ITS.  One  example  of  a  pedagogy  that  has  been  employed  is  cognitive  appren¬ 
ticeship.  This  pedagogical  approach  is  based  upon  allowing  the  student  to  experience  the 
most  difficult  aspects  of  a  cognitively  intense  task  where  assistance  is  available;  in  the  case 
of  an  ITS,  this  assistance  is  provided  by  an  intelligent  coach.  Another  widely  used  pedago¬ 
gy  is  guided  discovery.  Here,  the  student  is  provided  with  a  simulated  environment  in 
which  he  can  explore  various  concepts  and  conditions  through  the  manipulation  of  objects 
at  different  levels  of  abstraction.  The  level  of  intelligent  help  provided  may  be  minimal, 
such  as  a  tutor  that  presents  and  guides  the  proofing  of  some  hypotheses. 

To  date,  only  a  small  number  of  controlled  experiments  that  attempted  to  assess  the 
effectiveness  of  specific  systems  have  been  reported  in  the  literature.  Their  results  (summa¬ 
rized  in  Figure  2)  indicate  that  ITS  technology  is  a  potentially  valuable  field. 


Table  1.  Characteristics  of  Military  ITS  in  the  1980s 


ITS 

Knowledge  Base 

Student  Model 

Tutor 

SOPHIE  (Electronic 
maintenance) 

Semantic  network  with 
mathematical  model 

Overlay  onto  model 

Simulation  with  articulate 
expert 

STEAMER  (Steamship 
propulsion) 

Mathematical  model 
with  procedural  network 

Overlay  onto  model 

Graphically  simulated 
environment  with  advisor 

IMTS  (Corrective 
maintenance) 

General-purpose  correc¬ 
tive  maintenance  net¬ 
work 

Overlay  onto  hierar¬ 
chical  curriculum 
network 

Simulation  with  human 
instructor  interface  and 
concept  distance 

CBMS  (Memorization  of 
hierarchical  concepts) 

Weighted  semantic  net¬ 
work 

Overlay  onto  net¬ 
work 

Hints  linked  by  problem 
type 

TRIO  (Radar  intercept 
officer  tactics) 

Rule-based  representa¬ 
tion  with  goal-oriented 
problem  solver 

Overlay  onto  expert 
solutions 

Real-time  simulation  with 
articulate  expert  and  voice 
interaction 

QUEST  (Operation/trouble¬ 
shooting  of  electronic 
circuits) 

Rule-based,  “zero- 
order,”  and  “first-order” 
qualitative  models 

Overlay  onto  pro¬ 
gressively  sophisti¬ 
cated  model 

Simulation  with  problem 
generator  and  explicit 
queries 

SATS  (Radar  signal 
analysis) 

Rule-based  representa¬ 
tion  with  ordered  lists 

Overlay  onto  rule- 
based  representation 

Rule-based  simulation 
and  metatutor  with  prob¬ 
lem  generator 

MACH-III  (HAWK  missile 
maintenance) 

Hierarchical  network 

Overlay  onto  practi¬ 
cal  doctrine 

Simulation  with  articulate 
expert 

INCOFT  (PATRIOT  missile 
engagement  control) 

Rule-based  representa¬ 
tion 

Overlay  onto  expert 
solution 

Real-time  simulation  with 
articulate  expert 
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Figure  2.  Results  of  Previous  ITS  Evaluations 
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3.  STUDY  DATA 


While  the  study  did  not  conduct  an  exhaustive  survey  of  Government-sponsored 
ITS  R&D  efforts,  it  is  believed  that  the  majority  of  such  efforts  was  identified,  and  this  was 
enough  to  provide  insight  into  activity  in  this  area  as  a  whole.  A  total  of  47  projects,  spon¬ 
sored  by  16  different  Government  organizations,  were  studied.  These  efforts  addressed  the 
development  of  70  specifically  identified  ITS  or  ITS  authoring  environments. 

Table  2  summarizes  these  projects.  Grouped  by  sponsoring  organization,  this  table 
identifies  each  project  title,  the  duration  of  the  effort,  and  the  amount  and  type  of  funding. 
The  different  funding  types  are  grants  (either  Air  Force  Office  of  Scientific  Research 
(AFOSR),  the  DOEd  Fund  for  Improvement  of  Postsecondary  Education,  or  NSF),  SBIR 
procurement  contracts.  Technology  Reinvestment  Project  awards,  and  monies  appropriated 
from  Congress  for  R&D  work  (6.1  category  for  basic  research,  6.2  category  for  concept 
development,  6.3  category  for  engineering  development,  or  6.4  category  for  productiza¬ 
tion.)  For  each  project,  the  major  R&D  focus  is  described,  and  expected  ITS-related  prod¬ 
ucts  identified. 

Further  information  on  these  projects  can  be  found  in  a  companion  document 
[Youngblut  1994],  with  the  exception  of  the  NSRF-sponsored  development  of  STEAMER- 
GT  for  which  information  only  recently  became  available. 


Table 


Primary 

Sponsor 

Project  Title 

Start/ 

End 

Years 

Funding 

f 

Type 

$ 

AL 

Training  Research  for  Automated  Instruction 
(TRAIN) 

91-97 

AFOSR 

$3.6M 

Understand  the  relationship  am 
ological  variables,  and  instructi 

Fundamental  Skills  Training  (FST) 

90-96 

6.2 

$4.5M 

Research,  develop,  evaluate,  an 
schools  and  industry  under  Fed 

Intelligent  Computer- Assisted  Training  Testbed 
(ICATT) 

89- 

6.3 

S5.3M 

Create,  demonstrate,  and  evalu; 
and  deliver  intelligent  simulate 
related  tasks 

Basic  Job  Skills  (BJS) 

90-95 

6.2,  6.3 

S4.5M 

Develop  technology  and  metho 
nents  of  skills  that  define  high-' 
accelerate  skill  acquisition  and 

Intelligent  Training  Development  Methodologies 

93-95 

6.2 

$400K 

Identify  principles  for  merging 
and  development  tools  for  use  i 

improved  lii>r 

llllll 

;||  llll 

111! 

prototype  (nsfrwttanal  $yste> 

Virtual  Interactive  Techrtoldgtes  ITS  Develop- 
mwit  Sy&ein  (V  TVlf>£) ; : 

mm 

EstfawJ  findings  k  training  p. 
netd  (o  produdng  effick-tttlf 

:  ::  %  lllllllll 

Planned  effort 


le  2.  Summary  of  ITS  Data 


Focus 

ITS -Related  Products 

ITS  and  ITS  Authoring  System  Details 

Product  Type 

Target  Domain 

unong  automated  pedagogies,  person- 
ctional  effectiveness 

Human  performance  capabilities  taxonomy,  meth¬ 
ods  for  instruction- targeted  task  analysis,  auto¬ 
mated  instruction  development  guidelines 

Not  applicable 

Mathematics  Abstract  Reasoning  Tutoring  Archi¬ 
tecture  (MARTHA) 

Research 

Algebra  word  problem-solving 

Puzzler  Toolkits 

Research 

Scientific  concepts 

Solver  ITS 

Research 

Word  problem  solving 

Spreadsheet  ITS 

Research 

Problem-solving  with  spreadsheets 

Virtual  Interactive  System  for  Training  Applica¬ 
tions  (VISTA)  Laboratory 

Research 

Console  operations  and  spatial  navigation 

Loader  ITS 

Research 

Remote  crane  control  arm  operation 

Console  Operations  Tutor  (COPTER)  ITS 

Research 

NASA  shuttle  missions 

Phoenix  ITS 

Research 

Instrument- only  landing  in  the  F-16  aircraft 

and  transfer  ITS  technology  to  public 
ederal  technology  transfer  guidelines 

Word  Problem  Solving  T\itor 

Prototype 

Word  problem  solving 

Reading/Writing  in  a  Supportive  Environment  (R- 
WISE)  Tutor 

Prototype 

Basic  writing  skills 

Instruction  in  Scientific  Inquiry  Skills  (ISIS)  TXitor 

Prototype 

Scientific  inquiry  skills  for  biology 

luate  a  capability  to  rapidly  develop 
ttion-based  training  for  equipment- 

Microcomputer  Intelligence  for  Technical  Training 
Writer  (MITT)  Authoring  Shell 

Practical  use 

Troubleshooting  complex  devices 

Rapid  ITS  Development  System  (RIDES) 

Practical  use 

Interactive  system  behaviors 

STEP  Writer  Authoring  Environment 

Practical  use 

Procedural  skills 

hods  to  examine  cognitive  compo- 
h-tech  maintenance  expertise,  and  to 
id  foster  skill  generality 

Avionics  Troubleshooting  ITS  (Sherlock  II) 

Research 

F-15  navigation  electronics  troubleshooting 

F-15  Hydraulics  ITS  (HY-DRIVE) 

Prototype 

F-15  hydraulics  troubleshooting 

F-15  Comm-Nav  Penetration  Aids  ITS 

Prototype 

F-15  comm-nav  penetration  troubleshooting 

F-15  Radar  ITS 

Prototype 

F-15  radar  troubleshooting 

ig  principles  of  instructional  design 
;e  in  VR-based  simulations 

Software  Development  Toolkit  for  Development 
of  VR-based  ITS 

Prototype 

Developing  and  delivery  of  VR-based  ITS 

Orbital  Mechanics  ITS 

Research 

Orbital  Mechanics 

of  3'S^IN  results  t>y  implementing 
terns  l«t  4j|  ^rce<iomtan^  ■ 

(Joteniial  of  VRs,  while  addressing 
Ifutrirtg  rtevelnpment  and  delivery 

IT§  miltmrtttg  shell 

Opfef«tS<m  Jrtuf  tn&tttfetwtice  of  complex 

Specification  for  operational  VR-based  shell  :  :  - 

Operation  anB  maintei^nce  b^complex 

Table  2.  Sumi 


Primary 

Project  Title 

Start/ 

End 

Years 

Funding 

F< 

Sponsor 

Type 

$ 

AF  HSC 

Maintenance  Skills  Tutor  (MST) 

90-99 

6.4 

S2.5M 

ITS  productization 

ITS  User  Support 

94- 

Not  applicable 

Review  ITS  development  efforts 
and  formulate  cooperative  effort 

AMC, 

MICOM 

ARPA  Intelligent  Tutoring  System 

93-94 

SBIR 
Phase  II 

$489K 

Investigate  the  use  of  a  new  faul 
more  effective  and  efficient  diag 

Neural/Expert  Based  Client-Server  Architecture 
for  Multi-Node,  Task-Sharing,  Expert-Instruction 
ITS 

93-94 

SBIR 
Phase  I 

$50K 

Investigate  the  application  of  hy 
edge-based  strategies  to  the  prot 

STRICOM/ 

ARPA 

Simulator/Simulation- Based  Intelligent  Tutoring 
Systems 

93-94 

SBIR 

Phase  I 

$99K 

Develop  instructional  frameworl 
tion,  elicitation-based  assessmen 

Case-Based  Reasoning  for  Simulation-Based 
Intelligent  Tiitoring 

93-94 

SBIR 
Phase  I 

$70K 

Investigate  the  use  of  case-based 
sent  the  example  problems  and  p 

Hybrid  Simulator-Based  Intelligent  Tutoring 
System 

93-94 

SBIR 

Phase  I 

$50K 

Investigate  combining  skill-base 
structured  knowledge-based  dia$ 

Icon-Based  Intelligent  Tutoring  System  Utilizing 
Fuzzy  Expert  Systems  and  Multi-Media  Gaming 
Simulations 

93-94 

SBIR 

Phase  I 

$50K 

Demonstrate  feasibility  of  fuzzy 
dent  performance,  and  gaming  s: 

ARI, 

AT  MRU 

Visual  Knowledge  Representation  in  Multi- Media 
CyberSpaces 

92-94 

6.1 

$500K 

Investigate  how  VR  technology 
immersion  and  applying  results  1 

Multilingual:  Advanced  Technologies  for  Master¬ 
ing  Foreign  Languages 

89-95 

6.2 

$2.5M 

Conduct  research  in  technology 
ing  and  extend  understanding  of 

ary  of  ITS  Data  (Continued) 


ITS- Related  Products 

ITS  and  ITS  Authoring  System  Details 

Product  Type 

Target  Domain 

F-15  Avionics  Off-Equipment  Manual  Test  Stand 
ITS 

Production 

F-15  Avionics  Off-Equipment  Manual  Test 
Stand  Troubleshooting 

F-15  Pneudraulics  ITS 

F-15  Pneudraulics  Troubleshooting 

F-I6  Pneudraulics  ITS 

F-16  Pneudraulics  Troubleshooting 

F-15  Avionics  A  Shop  ITS 

Production 

F-15  Avionics  A  Shop  Troubleshooting 

F-15  Avionics  C  Shop  ITS 

Production 

F-15  Avionics  C  Shop  Troubleshooting 

F-16  Avionics  A  Shop  ITS 

Production 

F-16  Avionics  A  Shop  Troubleshooting 

F-16  Avionics  B  Shop  ITS 

Production 

F-16  Avionics  B  Shop  Troubleshooting 

F-16  Avionics  C  Shop  ITS 

Production 

F-16  Avionics  C  Shop  Troubleshooting 

F-16  Engine  ITS 

Production 

F-16  Engine  Troubleshooting 

F-15  Environmental/Electrical  Systems  ITS 

Production 

F-15  Environmental/Electrical  Systems 
Troubleshooting 

F-16  Environmental/Electrical  Supply  ITS 

Production 

F-16  Environmental/Electrical  Supply 
Troubleshooting 

explore  additional  applications, 
to  satisfy  mutliple  training  needs 

Not  applicable 

solation  technique  in  providing 
>sis  of  student  deficiencies 

ARPA  ITS 

Prototype 

Avenger  Table  Top  Trainer 

id  neural  network  and  knowl- 
:m  of  MITE  system  training 

MITE  ITS 

Proof- of-concept 

Task  allocation  for  team  members 

with  dynamic  scenario  regenera- 
multiple  knowledge  schemas 

MOS  91 B  Technical  Training  ITS 

Proof- of-concept 

Trauma  care 

easoning  to  capture  and  repre- 
3vide  student  explanations 

MOS  91B  Technical  Training  ITS 

Proof-of-concept 

Emergency  medical  procedures 

adaptive  training  techniques  and 
os  is  and  syllabus  modification 

MOS  91B  Technical  Training  ITS 

Proof- of-concept 

Ground  ambulance  evaluation  support 

MOS  63B  Technical  Training  ITS 

Proof-of-concept 

Maintenance  of  5-ton  trucks 

3gic  paradigms  in  analyzing  stu- 
tulation  techniques  for  learning 

Icon-based  ITS  with  two  data  bases  (63B  and  9 IB) 

Proof-of-concept 

5-ton  truck  oil  system  repair,  advanced  first- 
aid  skills 

m  be  used  as  a  tool  for  studying 
synthetic  training  environments 

Authoring  tools  for  construction  of  research  VRs 

Not  applicable 

!  enhance  foreign  language  train- 
Dreign  language  skill  acquisition 

Military  Language  Tutor 

Demonstration 

C3I  job-specific  language  skills 

Table  2.  Sum 


Primary 

Project  Title 

Start/ 

End 

Years 

Funding 

F 

Sponsor 

Type 

$ 

ARI, 

ARDA 

Intelligent  Flight  Trainer  for  Initial  Entry  Rotary 
Wing  Training 

93-95 

SBIR 

Phase  II 

$500K 

Integrate  ITS  methods,  adaptive 

thesis,  and  low-cost  simulation 

" 1 

DMSO 

Building  Education  and  Training  Technology  on 
Commercial  Software 

94-94 

6.3 

$400K 

Investigate  the  practical  feasibil 
oping  ESSCOTS  applications 

ONR 

Computer  Dialogue  Generation  to  Support 

Multiple,  Sophisticated  Tutoring  Tactics 

93-96 

6.1 

$380K 

Investigate  how  to  produce  inte 
dialogue  that  responds  to  user  n 

Participating  in  Reflective  Dialogues  in  a  Com¬ 
plex,  Real  World  Domain 

91-97 

6.1 

S370K 

Investigate  student  needs  for  re 
egies  for  choosing  what/how  in 

Development  and  Assessment  of  Alternative 
Tutoring  Strategies  for  an  Intelligent  Mathematics 
Tutor 

93-96 

6.2 

$523K 

Experimentally  study  effectives 
strategies  for  schema -based  sol 

Forecasting  Training  Efficiency  and  Transfer 
Effectiveness:  An  Automated  Cognitive 

Approach 

94-95 

6.2 

$160K 

Investigate  a  cognitive  analysis 
train  and  expected  degree  of  tra 

Uw|eepteltewmfr<)m  i»  Tutor;: 

9*96 

fci  ■ 

UtHity  of  Coaching  Models  and  Improved 
Measurement  Systems  lit  aft  ITS 

!!!!!!! 

S29SK 

Detet^ 

can  enhance  nindeltoj 

NPRDC 

Complex  Cognitive  Skills:  Objective  2 

92-95 

S375K 

Determine  the  feasibility  of  usi] 
appropriate  mental  models 

Linking  Cognitive  Styles  to  Instructional  Strate¬ 
gies  for  ITS 

93-94 

SBIR 
Phase  I 

$50K 

Develop  empirically  validated  1 
instructional  strategies,  and  inf< 

Damage  training  in  a  Virtual 

Environment 

94-9$ 

tilliili 

mil 

Demonstrate  ieetm|q«e$far  f 
training  in  a  virtoatenvfroaw 

NAWCTSD 

Low-Cost  Knowledge-Based  Tool  for  Rapid 
Prototyping  and  Development  of  ITS 

$50K 

Demonstrate  the  feasibility  of  a 
ment  of  courseware  exploiting 

Intelligent  Tutor  for  Electronic  Warfare  (EW)  Sit¬ 
uation  Assessment 

93-94 

SBIR 
Phase  I 

$272K 

Develop  a  standard,  modular  ar 
ligent  Embedded  Trainers  for  s! 

NSRF 

Enhancing  Naval  Reserve  Propulsion 

Engineering  Training 

93-95 

NSRF 

funds 

S600K 

Converting  STEAMER  to  simt 

SPAWAR 

Integrated  Training  Architecture  in  Support  of 
Active  Sonar  Systems 

92-94 

SBIR 

Phase  II 

$555K 

Design  and  demonstrate  a  pracl 
applicable  to  all  aspects  of  Sur 
tern  (SIJRTASS)  operator  profit 

ummary  of  ITS  Data  (Continued) 


Focus 

ITS-Related  Products 

ITS  and  ITS  Authoring  System  Details 

Product  Type 

Target  Domain 

)tive  training  techniques,  speech  syn- 
ion  technology 

Intelligent  Flight  Trainer  (IFT) 

Prototype 

Initial  Entry  Rotary  Wing  (IERW) 

sibility  and  cost  effectiveness  of  devel- 
ns 

ESSCOTS  applications 

Prototypes 

TBD 

intelligent  computer-generated  tutorial 
er  needs 

CircSim  Tutor 

Practical  use 

Negative  feedback  system  controlling  blood 
pressure 

A-Base 

Practical  use 

Physiology  of  acid-base  problems 

ir  reflective  dialogues,  identifying  strat- 
w  information  should  be  presented 

Prototype  explanation  generator  for  Sherlock 

Not  applicable 

liveness  of  alternative  instructional 

1  solving  of  algebra  word  problems 

Tutoring  in  Problem  Solving  Tutor 

Practical  use 

General  problem-solving  skills 

ysis  tool  to  support  forecasting  time  to 
if  training  transfer 

Extended  Cognitive  Analysis  Tool 

Not  applicable 

IpMcjhi:  relates  tutorial  actions  to  i 
ses  and  learning  effects  in  the  student 

Model  relating  tutorial  actions  to  resultant  coge; 
nitiy|  processes  and  learning  effects 

No!  applicable  ' 

a  f  ratoee  is  looking 

Bing,  and  the  value  ITS  compared  in 

Computer  Adaptive  Testing- Armed  Services 
Vocational  Aptitude  Battery  (ASVAB) 

Dtagflosis  of  system  problems  for  test ;  : 

admi  Jibrtrafors '  7 

r  using  an  ITS  as  a  means  of  training 

EW  cognitive  skills  training  tutor  requirements 
and  design  specification 

Prototype 

EW  cognitive  skills 

ted  links  between  cognitive  styles, 
information -presentation  methods 

Electronic  Warfare  Tutor 

i 

Prototype 

EW  tasks  in  Combat  Information  Centers 
(CICs)  on  Navy  ships 

or  integrating  damage  control  team 
onment 

ITi  %yfem  Specification* : j; 

Prototype 

ftamiige  7-f.i 

of  a  commercial  tool  for  rapid  develop¬ 
ing  both  CBT  and  ITS  technologies 

ITS  development  tool 

Proof- of-concept 

Domain  independent 

ITS  development  tool  design  specification 

Prototype 

Domain  independent 

ir  architecture  for  development  of  Intel- 
or  shipboard  EW  operators 

Electronic  Warfare  Intelligent  Embedded  Trainer 

Proof-of-concept 

EW  situation  assessment 

;imulate  a  LM-2500  gas  turbine  engine 

STEAMER-GT 

Production 

Marine  gas  turbine  propulsion 

tactical  application  of  ITS  technology 
Surveillance  Towed  Array  Sensor  Sys- 
roficiency 


Multistatic  Acoustic  Receiver  System  Trainer 


Demonstration 


STJRTASS 


Table  2.  Sum 


Primary 

Sponsor 

Project  Title 

Start/ 

End 

Years 

Funding 

I 

Type 

$ 

ARPA/ 

Technology 

Reinvest¬ 

ment 

Program 

Next  Generation  Authoring  Tools  and  Instruc¬ 
tional  Applications 

94-97 

TRP 

$6Ma* 

Accelerate  development  of  new 
applications 

Engineering  Academy  of  Southern  New  England 

94-96 

TRP 

$4Mb* 

Design,  implement,  evaluate,  ar 
of  education  for  engineers  in  th 

Retraining  the  Manufacturing  Workforce  for  the 
Biotechnology  and  Biomedical  Industries 

94-97 

TRP 

$1M* 

Develop  a  new  generation  of  er 
facilitate  transition  of  defense  v 
ical  engineering 

DOEd 

Computer  Lessons  for  Written  Harmony 

91-94 

FIPSE 

$150K 

Develop  software  for  the  cognil 
and  analyze  musical  exercises  i 

Computerized  Sight-Singing  Lessons  with 
Intelligent  Feedback 

91-95 

FIPSE 

$135K 

Develop  a  series  of  computer-b 
tional  teaching  in  sight-singing 

NASA 

Conduct  of  Fire  Trainer  (COFT)  Automation 

6.2 

$360K 

Automate  instruction  functions 
(COFT)  facility,  develop  tank  c 
integrate  with  simulation 

Early  Language  Intervention 

6.2 

$290K 

Adapt  the  Intelligent  Physics  Ti 
vention  for  children  with  disabi 

Horizontal  Integration  of  Battlefield 
Augmentation 

$1.1M 

Develop  an  ICAT  and  associate 
to  model  battlefield  informatior 

Assessing  the  Potential  of  Virtual  Realities  in 
Science  Education 

94-95 

$936K 

Chart  opportunities/challenges  < 
cognitive  trials  to  research  how 

Authoring  System  Development 

92-95 

$110K 

Develop  an  authoring  environm 
and  provide  a  tool  that  guides  tl 

Advanced  Training  Technologies 

87-96 

6.3 

$3M 

Enhance  effectiveness  of  NASA 
application  of  advanced  techno] 

NSF 

Methodology  for  Developing  a  Practical  Algebra 
Tutor 

92-95 

NSF 

Grant 

$790K 

Define  what  is  involved  in  takir 
and  using  it  to  implement  a  cou 

Using  Automatic  Speech  Recognition  To 

Improve  Reading  Comprehension 

NSF 

Grant 

$870K 

Test  feasibility  of  automatic  spe 
comprehension,  identifying  win 

Developing  Flexible  Explanation  Generators  for 
Intelligent  Tutoring  Systems 

NSF 

Grant 

$380K 

Develop  and  evaluate  a  flexible 
cally  generates  user- tailored  res 

Support  Systems  Based  on 
merclaMBt-the-Slielf  Software  '  §||§| 

mm 

111!!!! 

mill 

ill 

based  Jearnteg 

a.  With  matching  funds  from  the  contractor,  East/West  Consortium. 

b.  Universities  and  industries  that  belong  to  the  Academy  will  provide  an  addition  $8M  in  funds. 
*  Figure  reflects  funding  for  overall  project,  of  which  ITS  effort  is  only  a  part 
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Summary  of  ITS  Data  (Continued) 


Focus 

ITS- Related  Products 

ITS  and  ITS  Authoring  System  Details 

Product  Type 

Target  Domain 

>f  new  authoring  tools  and  instructional 

Intelligent  Tutor  Authoring  Tools 

Practical  use 

Customizable  domain  and  teaching  style 

Math  Tutor 

Basic  math  operations  to  beginning  algebra 

ate,  and  disseminate  an  integrated  system 
;  in  the  next  century 

Stamping  Tutor 

Design  for  Manufacturing  and  Assembly 

Injection  Molding  Tutor 

Practical  use 

Design  for  Manufacturing  and  Assembly 

1  of  engineering  teaching  environment  to 
ense  workers  into  biotechnology/biomed- 

Retraining  in  Biotechnology/Biomedicine  ITS 

Practical  use 

TBD 

cognitive  skills  needed  to  correctly  write 
;ises  in  four- part  harmony 

Written  Harmony  Tutor 

Practical  use 

Four-part  choral  style 

uter-based  lessons  to  supplement  tradi- 
nging 

Sight-Singing  Tutor 

Practical  use 

Sight-singing 

ctions  of  the  Conduct  of  Fire  Training 
tank  commander  and  gunner  system,  and 

COFT/ICAT 

Prototype 

M1A1  COFT  for  M-series  and  Bradley  tank 
gunnery 

sics  T\itor  to  support  early  language  inter¬ 
disabilities,  and  transfer  to  public  sector 

Early  Language  Intervention  System  (ELIS) 

Prototype  and 
production 

Basic  language  and  communication  skills 

ociated  communication  device  emulators 
nation  passing 

InterVehicular  Information  System  (IVIS)/ICAT 

Prototype 

IVIS  interface  operations  and  battlefield 
logistics 

nges  of  VR  for  science  learning,  perform 
i  how  VR  can  optimize  learning 

Virtual  Physics  Laboratory  (VPL)  II 

Prototype 

1-D,  2-D  kinematics,  statics,  and  dynamics 

ironment  derived  from  NASA’s  research 
ides  the  trainer  modeling  process 

ICAT  authoring  environment 

Prototype 

Training  based  on  simulating  work  environ¬ 
ment  and  modeling  a  personal  trainer 

VASA  training  through  development  and 
x;hnology 

ICAT  applications,  VR-training  applications,  and 
technology  transfer  to  Mission  Operations  Direc¬ 
torate.  Tools  for  building  and  maintaining  ICAT 
systems  and  complex  VRs.  Shared  VRs  for  collab¬ 
orative  training. 

Details  not  available 

i  taking  a  cognitive  tutor  methodology 
a  course 

Algebra  I  Tutor 

Prototype 

Algebra  1 

ic  speech  recognition  to  improve  reading 
ig  what  a  reading  coach  should  do 

Emily  Coach 

Research 

Reading 

cxible  explanation  facility  that  dynami- 
ed  responses  to  unanticipated  questions 

Biology  Tutor 

Prototype 

College-level  biology 

1  Issues  underlying  inquiry- 

4.  FINDINGS 


It  is  clear  that  ITS  R&D  remains  a  small  but  active  field.  Current  and  planned  pro¬ 
grams  ^are  funded  at  a  level  of  over  $56M  spread  over  a  14-year  period  from  FY87  to 
FYOO.1  This  figure  only  represents  a  snapshot  of  an  ever-changing  funding  profile,  howev¬ 
er,  since  efforts  completed  prior  to  FY94  are  not  included  and,  presumably,  additional 
efforts  may  be  initiated. 

As  is  often  the  case  when  investigating  R&D  efforts,  it  is  impossible  to  quantify 
what  has  been  accomplished  in  recent  years.  Even  though  a  few  ITS  are  coming  into  prac¬ 
tical  use,  there  is  a  current  lack  of  data  on  their  effectiveness.  Nonetheless,  the  analysis 

identified  several  themes  that  serve  to  give  a  feel  for  the  current  state  of  the  art  and  state  of 
the  practice. 

4.1  Activities  of  Different  Government  Organisations 

The  DOD  has  led  R&D  in  the  ITS  area  since  the  inception  of  this  technology  and 
continues  to  be  the  major  player  within  the  Government.  It  is  sponsoring  the  majority  of 
current  R&D  efforts,  both  large  and  small.  Using  the  snapshot  of  the  current  funding  pro¬ 
file,  the  proportion  of  funding  being  provided  by  different  Government  organizations  is 
shown  in  Figure  3,  together  with  details  of  the  breakdown  of  funding  within  the  DoD. 

Within  the  DOD,  the  Defense  Science  and  Technology  Reliance  Initiative  requires 
that  related  R&D  efforts  be  consolidated  and  co-located  at  a  single  site  [NPRDC 1 993]  The 
Air  Force  has  been  given  responsibility  for  R&D  in  Intelligent  Computer-Aided  Training, 
with  its  Armstrong  Laboratory  taking  the  lead  for  R&D  in  the  ITS  subarea.  To  help  fulfill 

this  responsibility,  Armstrong  Laboratory  has  developed  a  comprehensive  research  plan 
designed  to: 

Progress  from  laboratory  studies  in  artificial  tasks  toward  field  studies  of  fully 
implemented  ITS  for  real-world  tasks,  and 


is  pro-rated  over  the  duration  of  that  effort.  ach  case‘  For  each  program’  fundinS 
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Use  learning  theory  to  drive  further  research,  and  use  evaluation  data  to  con¬ 
strain  that  research. 


Finding  within  the  Government  Breakdown  of  BOD  Funding 


Armstrong  Laboratory  also  is  coordinating  with  the  Air  Force  Human  Systems 
Center  to  productize  ITS  for  Air  Force  use.  To  increase  efficiency  and  reduce  overlap,  the 
Reliance  Initiative  intends  that  other  DOD  groups  only  look  at  application  of  the  technolo¬ 
gy  with  respect  to  their  special  concerns.  Some  groups  are  following  this  guidance,  for 
example,  ARI  ARDA  is  investigating  the  development  of  an  intelligent  flight  trainer.  Oth¬ 
ers  are  looking  at  integrating  ITS  and  other  evolving  technologies,  for  example,  the  focus 
of  ONR’s  current  ITS  work  is  on  integrating  natural  language  interaction  capabilities  into 
ITS.  Several  other  groups,  however,  are  conducting  what  appears  to  be  ITS  basic  research 
independent  of  Armstrong  Laboratory,  although  none  of  the  current  efforts  seem  to  be  over¬ 
lapping  significantly  and  may  provide  useful  technology  advances  [Youngblut  1994]. 
Together,  Armstrong  Laboratory  and  the  Air  Force  Human  Systems  Center  are  responsible 
for  approximately  60%  of  the  ITS  R&D  funded  by  the  DOD. 

Another  DOD  initiative,  the  Technology  Reinvestment  Project,  is  also  addressing 
ITS  R&D.  Here,  ARPA  is  sponsoring  three  efforts  that  touch  on  ITS,  exploiting  dual  use 
potential  and  promoting  the  transfer  of  the  technology  into  specific  practical  applications. 

Other  than  the  DOD,  there  only  are  a  few  Government  organizations  active  in  this 
area.  NASA  is  veiy  interested  in  the  technology,  sponsoring  several  large  efforts  and  prom¬ 
ising  to  be  a  leader  in  the  research  into  virtual  reality  ITS  applications.  As  part  of  its 


18 


Advanced  Technologies  Program,  the  NSF  is  sponsoring  a  few,  small-scale  efforts,  but  ITS 
R&D  is  not  a  major  area.  The  DOEd  has  two  small  efforts  funded  under  its  Fund  for 
Improvement  of  Post-Secondary  Education.  In  general,  the  DOEd  has  not  funded  software 
projects,  and  although  this  is  planned  to  change  in  FY95,  the  DOEd  expects  to  fund  com¬ 
puter-aided  instruction,  rather  than  ITS,  R&D  at  that  time. 

4.2  Advances  in  Basic  Technology 

Roughly  one-third  of  the  examined  efforts  are  directly  concerned  with  ITS  basic 
technology  development.  (The  term  basic  technology  development  is  used  to  distinguish 
efforts  looking  at  how  to  implement  or  extend  ITS  technical  capabilities  from  those  efforts 
primarily  concerned  with  developing  ITS  authoring  systems  or  bringing  the  technology 
into  practical  use.)  These  efforts  have  a  current  funding  profile  as  shown  in  Figure  4.  They 
cover  the  range  from  well-funded,  multi-year  projects  to  short,  small-scale  efforts.  The 
majority,  however,  fall  into  the  latter  category. 


Figure  3.  Funding  for  ITS  Basic  Technology  Development 


The  issues  being  addressed  range  from  investigation  of  dynamic  scenario  genera¬ 
tion  to  investigation  of  the  joint  application  of  hybrid  neural  networks  and  knowledge- 
based  strategies.  However,  there  are  two  areas  of  popular  interest.  The  first  is  the 
traditional  area  of  student  modeling  and  diagnosis  of  student  deficiencies,  the  second  is 
improving  ITS  communication  with  a  student  through  interactive  dialogue. 
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In  the  first  case,  in  their  Simulator/Simulation-Based  Intelligent  Tutoring  System 
project,  STRICOM  and  ARPA  are  sponsoring  researchers  to  investigate  student  assessment 
based  on  elicitation  techniques  and  to  investigate  integrating  multiple  knowledge  represen¬ 
tation  techniques  in  an  ITS.  Other  STRICOM-  and  ARPA-funded  efforts  are  looking  at 
using  integrated  neural  network  and  expert  system  strategies  for  modeling  and  diagnosis, 
and  at  using  expert  systems  analysis  and  control  techniques  and  fuzzy  logic  in  student  mod¬ 
eling.  AMC  is  funding  an  effort  looking  at  diagnosing  student  deficiencies  using  innovative 
fault  isolation  techniques.  A  planned  ONR  effort  is  intended  to  determine  whether  mea¬ 
sures  of  what  a  student  is  looking  at  can  enhance  student  modeling. 

Research  into  interactive  dialogue  between  an  ITS  and  student  offers  to  improve 
system  effectiveness  and  usability  in  several  ways.  There  are  three  interesting  projects  in 
this  area,  with  quite  different  focuses.  First,  ONR’s  Computer  Dialogue  Generation  project 
is  developing  a  capability  whereby  the  CircSim  Tutor  can  carry  out  several  different  types 
of  interaction,  involving  questions,  acknowledgments  (positive,  negative,  or  neutral), 
multi-part  hints  and  explanations,  and  summarizing  discourses.  These  can  be  used,  for 
example,  in  providing  directed  lines  of  reasoning  and  in  natural  language  dialogue  where 
the  tutor  explores  students’  errors  with  them,  probing  for  misconceptions.  In  a  separate 
effort,  ONR  has  funded  researchers  to  investigate  how  to  support  reflective  dialogues  by 
identifying  the  types  of  questions  students  want  to  ask,  identifying  strategies  for  choosing 
the  information  to  include  in  responses,  and  organizing  and  presenting  the  information  in  a 
useful  way.  The  third  effort  is  an  NSF  project  in  which  the  major  focus  is  on  providing  a 
flexible  explanation  facility.  The  system,  in  this  case  the  Biology  ITS,  will  dynamically 
generate  responses  to  unanticipated  questions  and  tailor  these  responses  to  individual  stu¬ 
dents;  it  will  also  allow  the  generation  of  what-if  questions  on  the  subject  matter. 

Of  course,  the  efforts  just  discussed  all  use  natural  language  processing  technology. 
Other  efforts  are  likewise  exploiting  this  technology.  ARI’s  Multilingual  project  is  using  it 
to  develop  skill  acquisition  in  reading,  writing,  listening,  and  speaking  in  both  Arabic  and 
Spanish.  The  Emily  Coach  is  using  this  technology  to  improve  reading  comprehension 
skills.  These  two  efforts,  and  STRICOM’s  Icon-Based  ITS,  are  the  only  ones  actively  using 
automatic  speech  recognition.  From  the  ITS  output  perspective,  ARI’s  Intelligent  Flight 
Trainer  and  NASA’s  Conduct  of  Fire  Trainer  project  are  using  speech  synthesis  to  deliver 
coaching  and  advice  to  students. 

A  rather  different  area  of  research  is  being  sponsored  by  DMSO  (and,  potentially, 
by  NSF).  Here  researchers  are  looking  at  exploiting  existing  commercial-off-the-shelf 
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(COTS)  software  products  for  training  aids.  They  are  looking  at  how  to  add  a  software 
“wrapper”  that  turns  the  application  into  an  education  or  training  environment.  This  work 
is  in  its  earliest  stages,  currently  identifying  which  COTS  can  be  used,  identifying  educa¬ 
tional  outcomes  they  can  affect,  developing  broad  principles,  and  identifying  standards  to 
guide  the  development  of  educational  wrappers.  Two  COTS  products  that  have  been  iden¬ 
tified  as  prime  candidates  for  use  as  the  basis  of  an  ESSCOTS  are  RDSS  (a  model  of  why 
groups  revolt)  and  SIMHEALTH  (a  simulation  for  studying  and  designing  alternative 
health  systems). 

Other  topics  being  addressed  by  isolated  efforts  include  dynamic  regeneration  of 
scenarios  based  on  diagnosis  of  student  knowledge,  and  the  use  of  case-based  reasoning  to 
represent  problems  and  structure  learning. 

In  an  effort  to  summarize  the  current  state  of  the  art.  Table  3  presents  the  key  char¬ 
acteristics  of  current  prototype,  demonstration,  and  practical  use  systems,  where  known. 
There  are  some  indications  of  a  maturing  technology,  in  particular: 

•  A  change  in  the  focus  of  on-going  work  from  ITS  component  and  construction 
issues  to  specific  technology-related  topics.  Comparatively  few  efforts  are 
engaged  in  basic  research  into  such  topics  as  student  and  expert  modeling; 
moreover,  these  efforts  are  all  small  scale.  Instead,  researchers  are  shifting  their 
attention  to  topics  such  as  interactive  dialogue  and  wider  tutoring  concerns. 

•  Development  of  standard  ITS  architectures.  Here  one  effort  is  developing  an 
architecture  for  a  specific  domain  (electronic  warfare)  and  NASA  is  refining  an 
existing  general  purpose  NASA  architecture. 

•  ITS  systems  and  ITS  authoring  systems  are  being  developed  as  research  tools 
to  support  investigation  into  non-ITS  issues.  For  example,  in  the  first  case,  the 
Emily  Coach  is  intended  to  support  investigation  into  the  technical  feasibility  of 
using  automatic  speech  recognition  to  improve  reading  comprehension.  In  the 
second  case,  several  tools  are  being  developed  that  will  rapidly  generate  ITS  for 
experimentation  in  virtual  environments. 

Apart  from  the  general  change  in  focus,  there  are  some  new  technology  areas  under 
investigation,  primarily  in  the  area  of  VR.  Here  researchers  are  attempting  to  lay  the  foun¬ 
dation  for  future  work  by  investigating  the  training  potential  of  virtual  environments  and 
developing  guidelines  for  merging  principles  of  instructional  design  and  development  tools 
for  use  in  VR-based  simulations.  Although  some  four  or  five  ITS  efforts  fall  into  this  cate- 
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gory,  these  efforts  are  small  and  do  not  represent  large  funding  investments.  Even  so,  this 
is  an  area  likely  to  experience  increasing  attention  in  coming  years. 

On  the  whole,  this  is  all  good  news.  Further  investigation  of  basic  technology  issues 
needs  to  wait  on  a  better  understanding  of  the  capabilities  of  already  developed  technology. 
The  researchers  need  feedback  from  the  use  of  actual  systems  that  will  provide  some  mea¬ 
sure  as  to  the  value  and  deficiencies  of  this  technology:  steps  must  be  taken  to  ensure  that 
this  vital  step  occurs.  Meanwhile,  researchers  are  focusing  on  specific  issues  worthy  of 
attention. 


4.3  Push  Towards  ITS  Technology  Transfer 

The  Air  Force  is  sponsoring  two  major  technology  transfer  efforts,  each  of  which 
has  a  formalized  production  approach.  The  Air  Force  Human  Systems  Center  is  focusing 
on  introducing  the  technology  into  practical  Air  Force  use,  while  Armstrong  Laboratory  s 
Fundamental  Skills  Training  project  is  directing  its  work  towards  the  public  sector.  Tech¬ 
nology  transition  is  expensive  and  current  funding  for  these  efforts  represents  roughly  12% 
of  the  total  ITS  funding,  as  illustrated  in  Figure  5. 
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Figure  4.  Funding  for  ITS  Technology  Transfer 
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Table  3.  Chai 


ITS 

Knowledge  Representation 

Student  M 

Word  Problem  Solving  Tutor 

Declarative  and  procedural  representations 

Separate  declarative  and  pro 
tions 

Reading/Writing  in  a  Supportive  Environment 
(R-WISE)  Tutor 

Semantic  network  using  a  system  of  nodes  (con¬ 
taining  both  content  and  attributes)  and  links 
(typed  to  show  relationships) 

A  rule-based  model  is  used  tc 
A  second  model  collects  fine 
information  on  student  activi 

Instruction  in  Scientific  Inquiry  Skills  (ISIS) 
Tutor 

Knowledge  is  stored  in  the  context  of  the  simula¬ 
tion  and  text  files  for  students  to  study 

Endorsement-based  student  i 
declarative  and  procedural  n 

Sherlock  family 

Object-oriented  approach 

Uses  set  of  standards  for  effs 
ing,  matching  student  behavi 

ARPA  ITS 

Production  rule-based  and  dependency  model- 
based  approaches 

Model  is  copy  of  domain  hie 
training  attributes,  fault  isola 
approach  based  on  separate  d 
testing  with  a  sub-scenario 

Multi-Node,  Task-Sharing,  Expert- Instruction 
(MITE)  ITS 

Integration  of  neural  network  and  knowledge- 
based  strategies 

MOS  9 IB  Technical  Training  ITS  (Simulator/ 
Simulation-Based  ITS  Project) 

Integrated  causal  model  knowledge  structure 
(INKS)  combining  five  representation  schemas 
(scripts,  object  frames,  semantic  nets,  production 
rules,  mental  models) 

Assessment  based  on  hybrid 
compiled  into  INKS  framew 

MOS  9 IB  Technical  Training  ITS  (Case-Based 
Reasoning  for  Simulation-Based  ITS  Project) 

Object-oriented  representation  of  cases  with  case- 
based  reasoning  supporting  automatic  knowledge 
acquisition  and  presentation 

Contains  student  actions  and 
principles,  procedures,  and  t 
and  those  mastered 

MOS  63B  and  MOS  9 IB  Technical  Training 

ITS  (Hybrid  Simulator-Based  ITS  Project) 

Explicit  cognitive  models  implemented  by  algo¬ 
rithmic,  neural  network,  or  expert  system 
approaches 

Cognitive  model  with  “sites’ 
ences  with  expert,  structured 
diagnosis  with  deficiencies  e 
rule-,  or  knowledge- based  le 
cies  model 

Icon-based  ITS  (MOS  63B  and  MOS  91B) 

Multi-media  database,  fixed  expert  rule  set  and 
data  array,  modifiable  expert  rule  set  and  data  array 

Uses  fuzzy  logic  expert  sysfi 
niques.  Supported  by  person 
data  base. 

Military  Language  Tutor 

Rule-based 

Rule-based  model  with  bran< 
remediation  amount  and  levs 

Intelligent  Flight  Trainer  (IFT) 

Hybrid  algorithmic,  neural  net,  expert  system 
approach 

Infers  student  proficiency  on 
performance  and  known  pile 

CircSim  Tutor 

Object-oriented  knowledge  base  with  three  levels 
of  detail 

Combination  of  overlay  and 
approaches  to  model  knowle 
tions,  certainty  factor  to  mea 

haracteristics  of  Current  ITS 


it  Model 

Tutoring  Strategy 

User  Interface 

1  procedural  representa- 

Coach  practice  environment 

— 

GUI 

ed  to  drive  the  instruction. 

;  fine-grained,  historical 
tctivities  for  reporting 

Acts  as  cognitive  partner,  using  three  separate  tutors, 
hybrid  paradigm  combining  adaptive  tutoring  and  reifica¬ 
tion 

— 

GUI  with  animation 

lent  model  using  both 
ral  representations 

Cognitive  apprenticeship 

Ecosystem  simula¬ 
tion  driving  sce¬ 
nario 

GUI 

'  effective  problem  solv- 
>havior  against  these 

Cognitive  apprenticeship  through  intelligent  coached  prac¬ 
tice  and  intelligent  reply,  expert  teaching  specified  by  rules 

Device  simulation 
driving  scenario 

GUI  with  windows,  interactive  still 
video  for  equipment  representation, 
hypergraphics,  dynamically  updated 
block  diagrams 

1  hierarchy  annotated  with 
isolation  diagnostic 
ate  deficiencies  model  and 
rio 

Provides  observation,  diagnosis,  and  tutoring  modes  with 
scenario  generation  based  on  student  performance 

Weapon  simulation 
driving  scenario 

GUI  supported  by  intelligent  UI  design 
responsible  for  processing  student 
inputs,  and  presenting  scenario  dis¬ 
plays 

Information  not  available 

brid  elicitation  techniques 
nework 

Instruction  driven  by  models  of  needed  thinking  skills. 
Dynamic  scenario  regeneration  based  on  student  knowl¬ 
edge 

Human  body  simu¬ 
lation  for  examples/ 
exercises 

Animation 

and  decisions,  details  of 
nd  techniques  presented 

Presents  examples,  principles  and  methods,  and  compares 
student's  solutions  to  stored  solutions.  Using  student  input 
to  “learn”  which  examples  are  helpful 

Human  body  simu¬ 
lation  for  examples/ 
exercises 

Text  and  graphics  with  animation, 
video,  pictures,  hypertext 

ites”  for  capturing  dififer- 
ured  knowledge-based 
ies  evaluated  in  skill-, 
id  level,  separate  deficien- 

Skill-based  adaptive  training  techniques  driven  by  identi¬ 
fied  deficiencies 

Vehicle  simulation 
driving  scenario 

Object-oriented,  menu-based  GUI 
with  virtual  displays  of  controls  and  the 
HCI,  visual,  auditory,  tactile,  and  prio- 
prioceptive  interfaces,  voice  synthesis, 
multi -media 

systems  analysis  tech- 
rsonal  characterization 

Three  stages  of  learning  based  on  gaming  simulation  tech¬ 
niques  (tutorials  and  drill-and-practice,  simulation,  interac¬ 
tive  educational  gaming),  with  dynamic  reconfiguration  of 
gaming  simulations,  decision  trees  for  tutoring  strategies 

Gaming  simulation 
driving  scenario 

Menu-based  with  icons  using  graphics 
and  multi-media  presentation  tech¬ 
niques,  windows 

iranching  logic  to  control 
level 

Lessons  composed  of  exercises  of  14  types 

— 

GUI  with  windows,  supported  by  text- 
based  natural  language  interactive  dia¬ 
logue,  animation,  sound 

y  on  basis  of  observed 
pilot  capabilities 

Adaptive  training  techniques  with  evaluator/helper  and 
aided  control  to  fly  simulator,  and  advice  via  synthesized 
speech  output  capabilities 

Vehicle  simulator 
driving  scenario 

Multi-modal  cueing  system  (visual, 
motion,  proprioceptive,  auditory,  tac¬ 
tile)  for  student,  GUI  for  instructor 

and  bug  library 
iwledge  and  misconcep- 
tneasure  change  over  time 

Dynamic  multi-level  planner,  using  hierarchically  orga¬ 
nized  rules.  Multi -part  hinting  and  explanations,  directed 
lines  of  reasoning,  and  other  interactive  dialogue,  strategies 
under  experimentation 

Interactive  natural  language  input/out¬ 
put,  text- based,  windows 

ITS 


Tutoring  in  Problem  Solving  (TiPS)  Tutor 


Electronic  Warfare  Tutor 

Electronic  Warfare  Intelligent  Embedded  Trainer 


STEAMER-GT 


Multistatic  Acoustic  Receiver  System  (MARS) 
Trainer  Demonstration  System 

Written  Harmony  Tutor 


Sight-Singing  Tutor 


Conduct  of  Fire  Training  (COFT)/ICAT 


Early  Language  Intervention  System  (ELIS) 


Inter  Vehicular  Information  System  (IVIS)/ICAT 


Virtual  Physics  Laboratory  (VPL)  II 


Algebra  I  Tbtor 


Biology  Tutor 


Table  3.  Characteris 


Knowledge  Representation 

Student  M 

Object-oriented  database  of  problems  and  solu¬ 
tions  using  multiple  schemas  linked  together 

Incorporates  fuzzy  logic,  \oc< 
dent  actions  using  linked  sch 
provides  long-term  picture  ol 

Rule-based,  including  process  model  of  inductive  and 
deductive  decision-making 

Developed  through  monitoring 
use  of  interruption  analysis 

Mathematical  model  interfaced  with  dynamic  graphi¬ 
cal  objects 

Not  develo 

Tree-structure  of  related  goals,  concepts,  procedures, 
etc. 

Overlay  on  domain  knowledge 

Rule-based  using  hypertext  stack 

Response  matrix  of  questions  £ 
given 

Rule-based  using  hypertext  stack 

Response  matrix  of  questions  £ 
given 

Production  rules 

Descriptive  approach  based  on 
hierarchically  organized 

Hybrid  (production  rules,  decision  trees  and/or  sche¬ 
mas 

Descriptive  approach  based  on 
hierarchically  organized  model 
teacher  guidance 

Hybrid  (production  rules,  decision  trees  and/or  sche¬ 
mas 

Descriptive  approach  based  on 
hierarchically  organized 

Production  rules  and  mixture  of  other  schemas 

Descriptive  approach  based  on 
solving  procedures  used  by  stu 
actions  w  ith  patterns  of  actions 
(in)appropriate  to  indicate  erro 
vide  teacher  guidance 

Production  rule  model 

Relates  student  behavior  to  so: 
duction  firings  in  the  cognitive 

Structured  around  viewpoints  (descriptions  of  objects 
or  processes)  and  models  (built  from  viewpoints  and 
supporting  simulation  and  visualization) 

Overlay  subset  of  know  ledge  b 
around  viewpoints  and  models 

‘eristics  of  Current  ITS  (Continued) 


nt  Model 

Tutoring  Strategy 

Type  of 
Simulation 

User  Interface 

local  model  records  stu- 
d  schemas,  global  model 
ure  of  student  knowledge 

Cognitive  apprenticeship  with  three  stages  (demonstrator/ 
open  system  expert,  collaborator/Socratic  teacher,  coach / 
cheerleader),  “look-back”  facility 

— 

Menu-based  graphics,  windows,  voice 
recording 

TBD 

"T - - - - - - - - - 

oring  student  behaviors  and 
is 

piree  modes:  of  operation:  free-run,  interactive  training,  train¬ 
ing  debrief  & 

— 

Graphics  with  digital  maps 

eveloped 

Via  associated  workbooks 

Marine  gas  turbine 
engine 

GUI  representing  five  primary  control 
panels,  supported  by  sound  synthesis 

ledge  base 

Cognitive  apprenticeship  with  teachers  by  alternating  between 
diagnostic  and  training  processes 

For  MARS  device 

Text  and  graphical  GUI 

ions  asked  and  answers 

Progress  through  exercises  based  on  student  mastery,  individu¬ 
alized  multi-layer  feedback  for  errors,  exercise  generator 

— 

Menu-based  GUI,  supported  by  sound 
synthesis 

ons  asked  and  answers 

Progress  through  exercises  based  on  student  mastery,  individu¬ 
alized  multi-layer  feedback  for  errors,  exercise  generator 

— 

Menu-based  GUI,  supported  by  sound 
synthesis  and  pitch  detection  for  voice 
input 

id  on  historical  modeling, 

Based  on  trainer-modeling”  and  just-in-time  coaching,  pro¬ 
vides  after-action  reviews,  and  scenario  generator 

COFT  simulator 
driving  scenario,  and 
simulations  of  crew 
members 

GUI  w  ith  virtual  display  of  controls,  etc. 

d  on  historical  modeling, 
lodel  used  to  provide 

Initial  screening  of  capabilities,  then  displays  pictures  and 
questions  student  about  objects  in  the  pictures 

— 

GUI  with  multi-media  presentations, 
speech  recognition/synthesis 

d  on  historical  modeling, 

Based  on  trainer-modeling”  and  just-in-time  coaching 

Communication 
device  emulations 

Menu-based,  touchscreen  monitor,  ani¬ 
mation  and  digitized  panels  to  represent 
equipments,  sound  output 

d  on  capturing  problem- 
*  student,  patterns  of 
ions  classified  as 
errors,  model  used  to  pro- 

Intelligent  coaching  through  experimentation,  supported  bv 
visualization 

For  virtual  labora¬ 
tory  environment 
and  simulated  work 
environment 

VR  interface  providing  visual  and  audi¬ 
tory  displays,  tactile/force  feedback, 
motion  tracking  (uses  eyephones,  data 
glove,  chair  equipped  with  hand  control¬ 
lers  for  3-D  orientation  and  translation 

3  some  sequence  of  pro- 
tive  model 

Model  tracing  approach  and  reification,  ACT-R  cognitive  the¬ 
ory.  Tutor  is  a  learning  environment  in  which  helpful  informa¬ 
tion  can  be  provided  and  useful  problems  selected. 

— 

Text  and  graphics,  employing  spread¬ 
sheets,  symbol  manipulation  packages 
and  graphing  routines 

ge  base,  so  structured 
iels 

Answers  questions  with  individualized,  interactive  dialogue 

For  biological  pro¬ 
cesses 

Text  and  graphics,  initial  use  of  still  video 
and  animation 

To  reflect  the  Navy’s  phasing  out  of  steam  ships,  the  Naval  Surface  Reserve  Force 
is  sponsoring  NPRDC  to  convert  STEAMER  to  support  instruction  in  gas  turbine  propul¬ 
sion.  The  resulting  product,  STEAMER-GT,  will  be  based  on  the  mathematical  model  of 
STEAMER  but  include  additions  such  as  linking  to  an  Interactive  Technical  Manual. 
STEAMER-GT  will  be  used  to  replace  STEAMER  at  60  sites,  including  the  Gas  Turbine 
School  for  Navy  ratings.  While  not  specifically  addressing  technology  transfer  issues,  this 
effort  is  included  in  this  category  because  it  is  directly  concerned  with  bringing  an  ITS- 
related  product  into  actual  use. 

NASA’s  Intelligent  Physics  Tutor  has  been  licensed  to  a  private  company  for  com¬ 
mercialization.  In  addition,  two  of  NASA's  other  R&D  efforts  are  addressing  technology 
transfer,  though  this  is  not  the  major  focus  of  either  of  the  programs  concerned.  Its  Early 
Language  Intervention  Project  plans  to  transfer  NASA  technology  to  the  private  sector.  The 
Advanced  Training  Technologies  Project  is  focusing  on  transferring  technology  within 
NASA,  namely  from  the  Johnson  Space  Center  Software  Technology  Branch  to  the  Mis¬ 
sion  Operations  Directorate. 

With  the  current  state  of  ITS  technology,  this  level  of  focus  on  technology  transfer 
is  probably  appropriate.  Hopefully,  more  data  from  formative  evaluations  of  ITS  effective¬ 
ness  will  become  available  in  the  near  future  to  indicate  what,  if  any,  additional  funding  is 
required. 

4.4  Popular  Application  Domains 

For  those  ITS  systems  which  are  documented  as  under  development  for  practical 
use,  the  predominant  application  domain  remains  troubleshooting  of  electronic  devices.  In 
the  case  of  the  22  ITS  cited  as  intended  for  practical  use,  1 1  address  troubleshooting.  Of  the 
four  authoring  systems  cited  as  intended  for  practical  use,  only  one  is  specifically  intended 
for  the  development  of  ITS  for  troubleshooting  electronic  devices;  most  of  these  authoring 
systems,  however,  are  intended  to  be  produce  ITS  customized  to  various  domains,  and 
many  of  them  could  be  used  for  the  troubleshooting  domain.  Figure  6  identifies  the  intend¬ 
ed  application  domains  of  all  these  systems  under  development  for  practical  use. 

Of  the  remaining  proof-of-concept,  research,  and  prototype  ITS  being  developed, 
there  are  34  with  known  application  domains.  These  domains  span  a  wide  range.  The  most 
common  domain  is  equipment  operation  and  maintenance,  followed  by  troubleshooting  of 
electronic  devices  and  mathematics  (this  category  includes  algebra  word  problem  solving 
applications).  More  details  about  intended  application  domains  are  given  in  Figure  7.  As 
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appropriate  for  each  application  domain,  target  audiences  range  from  first-grade  school  stu¬ 
dents  to  operators  of  complex  equipment,  such  as  pilots. 


ITS: 


Electronic  device  troubleshooting  11 

.-P; : ...2 \ 

Moslc  '  2 

Biology/biotechnology  ,p.  2 

^Mathematics  pi  2 

Gas-turbine  propulsion  ?  ;t 

Early  language  intervention '  |§|||.jj:  1 
Medicine  |||J  l 


Authoring  Systems: 

.  ■  Electronic  device  troubleshooting  K  l 
Procedural  tasks  '  1 

Operating  complex  systems  1 

Unclassified  1 

Total  .  4 


Figure  5.  Application  Domain  Counts  for  Practical  Use  Systems 


ITS: 


::  lEqui^^ 

Mathematics  4  i 

Emergency  medical  care  f. p  3 
Basic  readi»$wrftlng^ 
Biotogy/biotechnoktgy  |§§  §g  t. 
Battlefield  logistics  I. 

Damage  control  g  I 

Orbital  mechanics  l 

Physics 

Problem  diagnosis  .......  H 

Spatial  navigation  §j§ 

Pliiki^ocatioii- ;  -  II® liSI  1 


Total  .7  :34 


Authoring  Systems: 

VR-based  research  2 

Is-  Unclassified  |§ ",  2 


Figure  6.  Application  Domain  Counts  for 
Proof-of-Concept,  Research,  and  Prototype  Systems 


Judging  from  this  data,  it  appears  that  researchers  are  starting  to  exploit  results  from 
the  last  30  years  of  investigation  into  electronic  troubleshooting  ITS  and  to  develop  systems 
for  practical  use.  Meanwhile,  attention  is  turning  to  a  broad  range  of  additional  domains. 
These  trends  are  encouraging,  providing  some  evidence  of  a  maturing  technology. 


4.5  Intelligence  of  ITS  Intended  for  Practical  Use 

An  interesting  question  is  how  much  intelligence  those  ITS  under  development  for 
practical  use  exhibit.  Over  half  are  derivatives  of  the  Sherlock  intelligent  coached  practice 
environment.  Sherlock  is  not  as  driven  by  a  dynamic  student  model  as  most  of  the  ITS  dis¬ 
cussed  in  this  work.  Its  intelligence  primarily  is  used  to  respond  to  student  requests,  rather 
than  to  intervene  directly. 

Of  the  non-Sherlock  based  ITS  under  development  for  practical  use,  information 
was  not  available  for  OMR’s  A-Base  tutor,  or  for  those  funded  under  the  Technology  Rein¬ 
vestment  project.  Like  its  predecessor  STEAMER,  STEAMER-GT  is  a  smart  simulation 
that,  currently,  does  not  use  intelligence  in  the  traditional  ITS  sense  (NPRDC  researchers 
hope  to  include  a  coach  in  STEAMER-GT  in  the  future,  but  such  an  effort  is  currently 
unfunded).  The  remaining  systems  to  consider  are  ONR’s  CircSim  and  Tutoring  in  Problem 
Solving  tutors,  and  NASA’s  Early  Language  Intervention  System,  all  of  which  provide  the 
traditional  types  of  cognitive  diagnosis  and  adaptive  remediation.  Only  CircSim  goes  fur¬ 
ther.  The  focus  of  this  effort  is  on  providing  additional  “intelligence”  through  interactive 
natural  language  dialogue  in  which  the  system  can  explore  their  errors  with  students  and 
discuss  issues  in  the  instructional  material  and  approach. 

Finally,  while  intelligence  is  not  a  question  pertinent  to  ITS  authoring  systems 
themselves,  it  would  be  useful  to  note  how  intelligent  the  ITS  developed  by  these  authoring 
systems  will  be.  Unfortunately,  this  information  is  not  available. 

4.6  Role  of  Training  Requirements 

Another  issue  is  how  ITS  developments  are  addressing  specific  training  require¬ 
ments.  Presumably  all  those  ITS  under  development  for  practical  use  are  intended  to  meet 
a  specific  training  requirement.  In  general,  however,  training  needs  are  being  met  in  one  of 
two  ways:  (1)  there  is  a  particular  training  need  that  has  driven  the  ITS  R&D  effort,  or  (2) 
a  training  requirement  has  been  chosen  primarily  to  provide  an  application  domain  in  which 
to  conduct  R&D.  Categorizing  the  ITS  systems  in  this  way  is  a  subjective  matter,  but  rough¬ 
ly  a  third  of  the  systems  fall  into  the  first  category,  predominantly  the  Sherlock  family  of 
tutors  and  STEAMER-GT,  with  the  remaining  two-thirds  being  in  the  second  category. 

It  is  interesting  to  note,  however,  that  those  cases  where  the  training  requirement 
can  be  said  to  largely  drive  the  research  do  not  correspond  one-to-one  with  systems  intend¬ 
ed  for  practical  use.  This  arises  because  a  few  systems  developed  as  research  vehicles  have 
come  into  practical  use,  and  because  some  efforts  intended  to  develop  practical  use  systems 
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are  building  generic  ITS  systems  (these  are  not  full  authoring  systems,  they  are  a  general- 
purpose  ITS  systems  which  can  be  supported  by  different  knowledge  databases).  Of  course, 
it  is  to  be  expected  that  meeting  a  specific  training  requirement  is  not  the  primary  driving 
force  where  ITS  are  developed  as  research  vehicles  for  investigating  particular  technology 
issues. 

It  is  important  to  note  that  only  Armstrong  Laboratory,  the  Air  Force  Human  Sys¬ 
tems  Center,  and  NASA  appear  to  be  specifically  trying  to  identify  the  training  needs  in  a 
broad  domain  and  look  at  how  ITS  could  meet  these  needs. 

4.7  Development  of  Authoring  Environments 

Another  relatively  new  area  of  investigation,  and  one  that  seems  likely  to  continue 
to  experience  growth,  is  that  of  ITS  authoring  environments.  These  are  systems  that  can  be 
used  to  build  a  new  ITS  from  scratch,  defining  both  instructional  approaches  and  content 
material.  Here  again,  the  funding  for  a  relatively  few  efforts  represents  a  significant  portion 
of  the  ITS  total  (very  roughly,  1 1%  of  efforts  are  allocated  32%  of  the  total  funding).  Figure 
8  shows  how  current  and  planned  funds  are  distributed  over  time. 


Figure  7.  Funding  for  ITS  Authoring  Environments 


Authoring  environments  are  largely  seen  as  a  mechanism  for  reducing  ITS  devel¬ 
opment  costs  and  transitioning  the  technology  into  practice.  For  example,  Armstrong  Lab- 


oratory  reports  that  a  “typical”  ITS  development  effort  takes  three  years  and  costs  around 
$1M,  and  its  Intelligent  Computer-Assisted  Training  Testbeds  project  has  the  goal  of  reduc¬ 
ing  tutor  development  time  by  80%  and  reducing  tutor  development  cost  by  up  to  95%.  As 
yet,  no  data  on  the  actual  cost-effectiveness  of  authoring  environments  has  been  reported. 
In  only  one  case,  Microcomputer  Intelligence  for  Technical  Training  Writer,  has  the  prod¬ 
uct  of  an  authoring  system  been  evaluated.  Here,  an  instructional  designer  and  subject  mat¬ 
ter  experts  used  Microcomputer  Intelligence  for  Technical  Training  Writer  to  develop  a 
tutor  to  train  students  in  troubleshooting  the  AN/UGC-141  Teletypewriter.  This  tutor  would 
be  used  to  supplement  classroom  instruction.  In  a  controlled  experiment,  researchers  found 
that  students  who  used  the  tutor  performed  significantly  better  in  a  paper  and  pencil  test 
than  those  that  did  not  use  the  tutor.  (On  average,  the  control  group  mean  score  increased 
23%  while  the  treatment  group  mean  score  increased  47%  [Parlett  1994].) 

It  is  also  interesting  to  note  that  authoring  environments  under  development  are 
split  almost  equally  between  prototypes  or  research  tools,  and  those  intended  for  practical 
use.  In  view  of  the  lack  of  data  about  the  benefits  of  these  environments,  this  is  evidence  of 
a  very  quick  move  into  the  development  of  practical  use  systems. 

Of  the  nine  authoring  systems  identified,  three  are  intended  to  support  research 
activities.  These  activities  fall  into  the  following  areas:  (1)  investigating  the  merging  of 
principles  of  instructional  design  and  development  tools,  (2)  looking  at  how  to  best  exploit 
the  capabilities  of  virtual  reality  technology  for  effective  training  of  cognitively  complex 
tasks,  and  (3)  determining  the  effect  of  immersion  on  training.  Three  more  authoring  sys¬ 
tems  are  essentially  packaging  of  traditional  ITS  approaches. 

The  remaining  three,  Armstrong  Laboratory's  Rapid  Intelligent  Tutoring  System 
Development  System,  the  Technology  Reinvestment  Project’s  Next  Generation  authoring 
system,  and  NASA's  Intelligent  Computer-Assisted  Training  authoring  system,  provide 
more  insight  into  the  approaches  being  used  to  develop  authoring  systems.  All  are  intended 
to  develop  ITS  for  customized  domains,  although  the  Rapid  Intelligent  Tutoring  System 
Development  System  is  constrained  to  situations  where  the  learner  can  benefit  from  inter¬ 
action  with  complex  computer-based  graphical  models.  It  is  not  clear  how  much  flexibility 
of  teaching  style  these  systems  will  support,  although  the  Rapid  Intelligent  Tutoring  Sys¬ 
tem  Development  System  allows  both  automatic  generation  or  custom-authoring  of 
required  instruction  and  the  developers  of  the  Technology  Reinvestment  Project  authoring 
system  are  emphasizing  a  customizable  teaching  style  yet  to  be  determined.  These  systems 
all  provide  the  user  with  graphical  editors  for  ITS  construction,  but  provide  different  para- 


digms  for  use  of  the  editors.  The  Rapid  Intelligent  Tutoring  System  Development  System 
approach  is  based  on  the  concept  of  objects  and  their  behaviors.  Though  the  full  process 
can  be  shortened,  ideally  the  Rapid  Intelligent  Tutoring  System  Development  System  usage 
starts  by  defining  a  set  of  learning  objectives.  The  user  then  develops  a  preliminary  design 
of  the  instruction  units  that  will  attain  the  objectives,  an  interactive  graphical  model  of  the 
domain  of  concern,  and  browsable  knowledge  units.  Finally  he  completes  the  instructional 
units.  The  Technology  Reinvestment  Project  authoring  system  essentially  operates  as  a 
knowledge  acquisition  shell  for  representing  and  rapid  prototyping  of  domain  and  tutoring 
knowledge.  With  its  Intelligent  Computer-Aided  Training  and  Tutoring  metaphor,  the 
NASA  approach  is  much  different.  Here,  developing  an  ITS  consists  of  building  a  simulat¬ 
ed  work  environment  and  providing  a  personal  trainer  that  uses  this  environment  to  instruct 
the  student.  Consequently,  the  authoring  system  integrates  a  simulation  package  for  work 
environment  construction  with  a  tool  that  guides  trainer  modeling,  that  is,  overlaying  a  sim¬ 
ulated  work  environment  with  lessons  and  providing  remediation  to  the  student. 

To  date,  the  authoring  systems  only  have  been  used  for  the  development  of  proto¬ 
type  ITS,  not  practical  use  ITS.  The  Microcomputer  Intelligence  for  Technical  Training 
Writer  has  received  the  greatest  usage,  with  nine  prototype  ITS  being  developed  for  appli¬ 
cations  ranging  from  acute  care  liver  transplants  to  a  cryptographic  unit. 

The  expectation  that  authoring  systems  will  help  to  transition  ITS  technology  into 
practice  is  based  on  the  premise  that  teachers,  and  instructional  developers,  will  be  more 
likely  to  use  ITS  that  they  themselves  have  developed  and/or  customized.  So  far,  there  is 
no  evidence  of  teacher  involvement  in  authoring  system  development,  even  for  those 
authoring  systems  intended  for  practical  use.  Unless  the  proper  consideration  is  paid  to  the 
needs  of  teachers,  authoring  systems  will  not  help  to  transition  the  technology. 

4.8  Evaluation  of  ITS  Effectiveness 

While  the  results  of  ITS  evaluations  to  date  (see  Chapter  2)  seem  promising,  there 
has  been  insufficient  experimentation  to  develop  any  general  conclusions  about  ITS  effec¬ 
tiveness  that  can  be  used  to  guide  ITS  usage  or  further  development.  Without  question,  this 
has  been  a  major  shortcoming  in  the  ITS  field.  There  are  some  difficult  issues  to  be 
addressed.  For  example,  an  ITS  is  a  complex  interaction  between  computer-based  technol¬ 
ogies  (such  as  artificial  intelligence,  human-computer  interaction,  and  language  process¬ 
ing),  pedagogy,  cognitive  science,  and  subject  matter.  How  can,  and  should,  the  effect  of 
these  different  concerns  be  addressed?  How  can  actual  training  effectiveness  be  assessed? 
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Basic  questions  as  to  what  should  be  measured  and  how  to  measure  have  yet  to  be 
answered. 

Armstrong  Laboratory  is  conducting  a  project  that  places  a  major  focus  on  investi¬ 
gating  these  issues.  The  goal  of  this  project,  the  Training  Research  for  Automated  Instruc¬ 
tion,  is  to  understand  the  relationship  among  automated  pedagogies,  personological 
variables,  and  instructional  effectiveness.  It  has  built  a  computer  laboratory  (called  the 
Cooperative  Lab,  or  Co-Lab)  to  serve  as  a  national  testbed  for  instructional  theory,  and  as 
a  resource  that  can  be  used  for  evaluating  ITS  in  a  controlled  laboratory  setting.  The  Train¬ 
ing  Research  for  Automated  Instruction  project  is  also  developing  and  promoting  wide¬ 
spread  use  of  a  standard  set  of  criterion  tests  that  are  designed  to  allow  benchmarked 
comparisons  of  instructional  approaches  and  so  help  diverse  researchers  share  data.  Other 
projects  that  include  some  (smaller- scale)  investigation  of  how  to  assess  ITS  instructional 
effectiveness  are  Armstrong  Laboratory's  Basic  Job  Skills  project,  the  ARI’s  Intelligent 
Flight  Trainer  project,  and  the  NPRDC's  Complex  Cognitive  Skills  project.  Additionally, 
investigating  ITS  effectiveness  in  training  transfer  is  one  of  the  goals  of  NASA’s  Intelligent 
Computer- Assisted  Training  Testbeds  Program. 

About  a  quarter  of  those  efforts  developing  an  ITS  have  performed,  are  performing, 
or  plan  to  perform,  formal  evaluations  of  their  products  (see  Table  4).  Here  again,  it  is  inter¬ 
esting  to  note  that  there  is  not  a  direct  correspondence  between  effectiveness  evaluation  and 
systems  intended  for  practical  use,  although  the  majority  of  such  systems  are  expected  to 
be  formally  evaluated.  The  primary  results  reported  from  completed  evaluations  are  as  fol¬ 
lows  (information  on  evaluation  results  was  not  available  for  Reading/Writing  in  a  Support¬ 
ive  Environment): 

•  Word  Problem  Solving  tutor — A  large-scale  field  evaluation  showed  that  stu¬ 
dents  exposed  to  traditional  instruction  improved  their  word  problem-solving 
performance  by  17%  over  a  school  year,  while  students  exposed  to  Word  Prob¬ 
lem  Solving  tutor  improved  by  26%  over  the  same  period  [Parlett  94]. 

•  Sherlock — Initial  field  testing  showed  that  20  to  25  hours  of  Sherlock  training 
produced  learning  equivalent  to  about  four  years  of  on-the-job  experience. 
More  recent  evaluations,  performed  with  a  criterion  of  real-world  performance 
of  the  most  difficult  part  of  the  job,  and  with  blind  scoring  of  performances, 
report  experimental  versus  control  effects  of  two  standard  deviations  [LRDC 
1993]. 
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•  Algebra  Tutor — Using  the  Iowa  Algebra  Aptitude  Test,  ITS-tutored  students’ 
scores  were  33.5,  as  contrasted  to  non-ITS  users’  scores  of  30.5.  Using  Scholas¬ 
tic  Aptitude  Tests,  the  average  score  of  ITS-tutored  students  was  410,  compared 
with  an  average  of  380  for  non-ITS  users  [Anderson  et  al.  1994]. 

•  Emily — Early  pedagogical  tests  showed  assisted  students  gained  a  reading  level 
higher  than  their  independent  reading  level  by  an  average  of  0.6  years.  Subjects 
misread  only  2.6%  or  the  given  words,  as  opposed  to  12.3%  when  no  assistance 
had  been  used  [Mostow  etal.  1994]. 

•  Virtual  Physics  Laboratory,  Physics  Tutor — Initial  evaluations  found  high  lev¬ 
els  of  student  satisfaction  with  the  tutor.  Evaluations  of  tutor  effects  on  course 
achievement  were  inconclusive,  but,  at  the  end  of  the  training  period,  tutored 
students  had  the  highest  scores  for  problem-solving  strategies  [Ross  and  Casey 
1992]. 


Table  4.  Efforts  Looking  at  Effectiveness  of  ITS  Products 


ITS 

Evaluation 

Status 

Word  Problem  Solving  Tutor 

Ongoing 

Reading-Writing  in  a  Supportive  Environment  Tutor 

Ongoing 

Science  Tutor 

Planned 

Sherlock 

Ongoing 

Intelligent  Flight  Trainer 

Ongoing 

CircSim  Tutor 

Ongoing 

Computer  Adaptive  Testing-Armed  Services  Vocational  Aptitude  Battery  Tutor 

Planned 

Electronic  Warfare  Tutor 

Planned 

Sight-Singing  Tutor 

Ongoing 

Written  Harmony  Tutor 

Ongoing 

InterVehicular  Information  System/Intelligent  Computer- Assisted  Training 

Planned 

Early  Language  Intervention  System 

Planned 

Virtual  Physics  Laboratory 

Ongoing 

Algebra  Tutor 

Ongoing 

Emily  Coach 

Ongoing 

Biology  Tutor 

Planned 

The  types  of  experiments  that  have  been  conducted  vary  widely  in  scope.  For  exam¬ 
ple,  a  pilot  evaluation  of  the  Word  Problem  Solving  tutor  employed  some  350  subjects, 
whereas  early  Emily  evaluations  have  been  limited  to  12  subjects.  The  systems  for  which 
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evaluation  data  is  available  vary  widely;  this,  together  with  the  lack  of  common  evaluation 
paradigms,  means  that  results  are  not  comparable  across  systems. 

Two  of  the  efforts  developing  ITS  authoring  systems  are  also  planning  formal  eval¬ 
uation  of  training  effectiveness,  specifically  the  efforts  underway  by  Armstrong  Laboratory 
and  the  Technology  Reinvestment  Project  Next  Generation  Authoring  Tools.  The  project 
plans  for  several  other  efforts  cite  plans  for  some  informal  evaluation,  but  this  evaluation 
is  likely  to  be  more  in  the  nature  of  the  small-scale  test  and  evaluation  routinely  required 
on  any  system  development  effort,  rather  than  addressing  the  particular  concerns  of  ITS 
technology. 

This  evidence  of  increased  attention  on  the  evaluation  of  ITS  effectiveness  is 
encouraging.  Since  most  evaluation  activities  have  yet  to  be  fully  defined  and  conducted, 
it  is  too  soon  to  say  whether  enough  attention  is  finally  being  paid  to  evaluation  issues. 

4.9  Focus  on  Pedagogy 

It  is  encouraging  to  note  that  several  efforts  are  looking  at  pedagogical  issues.  There 
are  no  major  trends  to  report,  but  rather  a  diffusion  of  attention  across  a  range  of  concerns. 
Some  of  this  work  is  closely  linked  with  the  technological  research  underway,  as  discussed 
previously  in  Section  4.2,  in  particular,  ONR’s  efforts  investigating  the  use  of  reflective  dia¬ 
logue.  Others  are  studying  particular  pedagogic  strategies,  notably  cognitive  apprentice¬ 
ship  and  inquiry-based  learning,  and  one  effort  has  the  stated  purpose  of  conducting 
experimental  studies  of  alternative  strategies.  There  are  four  efforts  focusing  on  cognitive 
issues:  (1)  identifying  stereotypical  patterns  of  cognitive  processes,  (2)  looking  at  linking 
an  individual’s  cognitive  style  with  methods  for  organizing  and  presenting  information,  (3) 
relating  tutoring  actions  to  resultant  cognitive  processes  and  learning  effects,  and  (4)  build¬ 
ing  tools  for  analyzing  cognitive  skills  and  mental  models.  Only  ARI’s  Multilingual  project 
work  with  pedagogy  is  primarily  driven  by  an  application  domain.  While  application 
domain  is  of  course  a  factor  for  the  other  efforts,  it  does  not  appear  to  be  the  driving  factor 
behind  the  work. 

Armstrong  Laboratory  and  NASA  have  a  couple  of  efforts  working  at  a  higher  lev¬ 
el,  endeavoring  to  assess  the  potential  that  VR  technology  has  for  instruction.  This  is  a  pop¬ 
ular  area,  not  only  within  the  ITS  community  but  the  computing  community  in  general. 

While  the  majority  of  the  current  projects  in  this  area  are  small,  short-term  efforts, 
there  is  some  reason  to  hope  that  this  focus  on  pedagogy  will  continue.  For  example,  Arm¬ 
strong  Laboratory’s  Co-Lab  intends  to  look  at  how  to  implement  and  assess  emerging 


knowledge  about  pedagogical  strategies,  and  it  has  another  large,  multi-year  project  inves¬ 
tigating  the  relationship  between  automated  pedagogies  and  instructional  effectiveness. 

4.10  Support  for  Teachers 

Two  groups  are  actively  including  teachers,  or  instructors,  in  their  work.  Supported 
by  NSF,  John  Anderson  has  been  working  closely  with  teachers  in  Pittsburgh,  Pennsylva¬ 
nia,  to  support  the  Pittsburgh  Urban  Mathematics  Projects.  As  part  of  its  Fundamental 
Skills  Training  project,  Armstrong  Laboratory  has  also  built  and  maintained  strong  ties 
with  local  schools,  involving  teachers  in  ITS  development  efforts  and  using  several  schools 
as  test  and  evaluation  sites. 

Some  other  efforts  are  looking  at  how  to  support  teachers  in  their  actual  use  of  an 
ITS.  NASA’s  previously  developed  Physics  Tutor  introduced  a  Teacher’s  Window  applica¬ 
tion  that  uses  the  underlying  student  model  to  provide  guidance  to  teachers  in  the  selection 
and  use  of  ancillary  instructional  aids.  A  similar  capability  is  being  developed  for  NASA’s 
ongoing  Early  Language  Intervention  Project.  The  TRP  authoring  tools  project  (Next  Gen¬ 
eration  Authoring  Tools  and  Instructional  Applications)  is  proposing  to  enable  instructors 
to  monitor  the  internal  state  of  the  tutor.  SPAWAR’s  Multistatic  Acoustic  Receiver  System 
Trainer  Demonstration  System  will  provide  various  reports  on  individual  and  group  perfor¬ 
mances,  giving  access  to  student-generated  comments  and  performance-based  parameters. 
Additionally,  this  ITS  can  generate  reports  that  reflect  on  the  strengths  and  weaknesses  of 
the  content  material. 

A  third,  and  final,  type  of  support  being  provided  to  teachers  is  the  provision  of  spe¬ 
cial  editors  and  tools  that  enable  them  to  prepare  their  own  lesson  materials.  (These  are  not 
full  authoring  environments  as  discussed  previously,  but  do  allow  some  customization  and 
extension  of  instructional  content.)  Included  in  this  category  are  the  Multistatic  Acoustic 
Receiver  System  Trainer  Demonstration  System,  the  Sight-Singing  and  Written  Harmony 
tutors,  and  the  Military  Language  Tutor. 

Nonetheless,  the  critical  issues  in  how  ITS  technology  should  best  support  teachers 
have  not  yet  been  systematically  identified,  much  less  understood.  This  area  requires  fur¬ 
ther  attention.  If  ITS  technology  is  to  mature  appropriately  and  come  into  widespread  prac¬ 
tical  usage,  the  special  needs  of  teachers  must  considered  as  an  integral  part  of  any 
development  effort. 
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5.  RECOMMENDATIONS 


ITS  technology  has  been  under  development  for  three  decades  and  there  are  signs 
that  it  is  starting  to  emerge  from  the  laboratory  and  come  into  practical  use.  While  there  still 
are  technology  issues  to  be  addressed,  the  emphasis  of  researchers  is  appropriately  starting 
to  shift  to  issues  pertaining  to  ITS  use. 

The  early  focus  on  general  ITS  components  and  construction  issues  is  changing. 
Today,  technology-related  R&D  is  looking  at  specific  topics,  such  as  flexible  explanation 
generation  and  dynamic  scenario  generation.  Several  of  the  efforts  are  looking  at  advancing 
those  parts  of  the  technology  whereby  an  ITS  exhibits  intelligence.  Apart  from  the  general 
change  in  focus,  there  are  some  new  technology  areas  under  investigation,  primarily  in  the 
area  of  VR.  On  the  whole,  activity  in  this  area  seems  promising  and  should  continue.  High¬ 
er  levels  of  funding  for  basic  technology  R&D  need  to  wait  on  a  better  understanding  of  the 
capabilities  of  already  developed  technology.  The  researchers  need  feedback  from  the  use 
of  actual  systems  that  will  provide  some  measure  as  to  the  value  and  deficiencies  of  this 
technology. 

Recommendation  1:  More  focus  is  needed  on  evaluation  issues.  The  next  several 
years  should  see  summative  evaluation  of  all  ITS  developed  for  practical  use,  while 
moving  towards  assessing  the  effectiveness  of  ITS  technology  in  general. 

Questions  such  as  What  is  the  potential  of  this  technology?  and  How  best  can  it  be 
used?  have  yet  to  be  answered.  To  date,  there  has  been  a  lack  of  formal,  or  even  informal, 
evaluations  that  work  toward  answering  these  questions,  so  it  is  encouraging  to  note  that  a 
number  of  current  efforts  seem  to  be  addressing  this  need.  This  trend  should  be  encouraged. 
A  single  organization  (in  view  of  the  Defense  Science  and  Technology  Reliance  Initiative, 
probably  Armstrong  Laboratory)  should  be  given  the  responsibility  to  collect  and  consoli¬ 
date  evaluation  data  produced  by  the  ITS  community  as  a  whole,  and  to  make  this  informa¬ 
tion  readily  available  to  feedback  into  new  development  efforts. 

Recommendation  2:  The  current  level  of  effort  in  the  area  of  authoring  systems 
should  not  be  expanded  at  the  present  time.  While  authoring  systems  can  be  valu- 


able  tools  for  researchers,  many  issues  need  to  be  addressed  before  more  such  tools 
are  developed  for  use  by  the  general  educational  community. 

The  question  of  support  for  the  development  of  authoring  systems  is  less  clear  cut. 
The  value  of  such  tools  for  bringing  ITS  into  widespread  use  will  be  substantial,  but  is  the 
time  right?  Authoring  systems  can  already  serve  as  valuable  tools  for  developing  ITS  that 
will  themselves  be  used  as  research  vehicles.  Without  more  data  on  the  general  effective¬ 
ness  of  ITS ,  however,  the  current  level  of  effort  may  be  premature.  Assuming  that  research¬ 
ers  do  establish  that  ITS  are  a  cost-effective  way  to  deliver  instruction,  will  the  typical  end 
user  be  able  to  use  an  authoring  system  to  produce  an  ITS  as  effective  as  one  developed  by 
experienced  researchers?  Is  there  any  way  in  which  the  quality  of  the  products  of  an  author¬ 
ing  system  can  be  predicted,  without  necessitating  formative  evaluation  of  each  individual 
product?  To  date,  authoring  systems  only  have  been  used  to  development  prototype  ITS,  so 
it  is  disturbing  to  note  that  half  the  authoring  systems  under  development  (or  planned  for 
development)  are  intended  for  practical  use  by  end  users  such  as  subject  matter  experts, 
instructional  developers,  and  teachers. 

Recommendation  3:  More  effort  is  needed  in  investigating  how  to  prepare  and 
support  teachers  in  their  use  of  ITS  technology. 

As  this  technology  continues  to  mature,  it  is  increasingly  important  that  the  needs 
of  teachers  are  understood.  This  includes  looking  at  what  ITS  products  teachers  actually 
want,  preparing  teachers  for  the  use  of  these  products,  and  ensuring  that  the  ITS  provide 
the  types  of  information  teachers  need  to  help  student  in  the  learning  process.  Should  teach¬ 
ers  receive  formal  training  prior  to  their  use  of  an  ITS,  a  practical  demonstration  of  the  sys¬ 
tem,  or  just  an  instruction  manual?  What  role  should  teachers  play  in  formal  evaluations  of 
ITS?  What  types  of  special  editors  and  tools  for  customizing  lesson  content  are  best  suited 
to  teachers’  use?  These  are  just  a  few  examples  of  the  many  issues  to  be  investigated.  With¬ 
out  due  consideration  of  teachers’  needs,  ITS  products  are  likely  to  end  up  as  shelfware. 

Recommendation  4:  The  role  of  ITS  in  supporting  new  instructional  approaches, 
such  as  learner  control,  collaborative  learning,  and  situated  learning,  should  be 
investigated  in  addition  to  VR  and  learning. 

A  related  issue  that  needs  studying  is  how  ITS  technology  can  support  the  changing 
vision  of  the  classroom.  Although  learner  control,  collaborative  learning,  situated  learning, 
and  virtual  reality  and  learning  have  been  identified  as  key  ITS  topics  for  the  1990s  [Shute 
and  Psotka  1994],  the  only  topic  receiving  significant  attention  from  researchers  in  the 


identified  efforts  is  virtual  reality  and  learning.  While  this  research  needs  to  be  continued, 
the  other  topics  merit  attention. 

Recommendation  5:  Continued  transition  of  the  technology  into  practice  use 
should  be  encouraged.  Public  awareness  of  the  potential  of  this  technology  needs  to 
be  increased. 

The  focus  starting  to  be  placed  on  transitioning  ITS  technology  into  practical  use 
should  be  encouraged.  Such  efforts  would  serve  multiple  purposes.  As  well  as  providing 
needed  instructional  tools,  increased  practical  use  of  ITS  would  provide  data  to  support 
assessing  the  effectiveness  of  this  technology  and  help  in  identifying  the  key  issues 
involved  in  better  supporting  teachers  in  their  use  of  the  technology.  Moreover,  some  tran¬ 
sition  activities  should  be  selected  as  demonstration  activities  used  to  raise  public  aware¬ 
ness  of  the  potential  of  ITS  technology.  In  developing  products  for  practical  use,  care  must 
be  taken  to  select  applications  where  this  technology  is  appropriate  and  to  satisfy  real  train¬ 
ing  requirements. 

The  importance  of  education  and  training  is  increasing,  both  in  Government  and 
industry.  This  is  in  response,  in  part,  to  such  drivers  as  the  increasing  need  for  a  flexible 
workforce  and  DOD  downsizing.  The  data  is  not  available  to  see  how  current  levels  of 
funding  differ  from  spending  in  previous  years,  but  declining  Government  and  DOD  bud¬ 
gets  are  likely  to  lead  to  substantial  funding  reductions  in  future  years.  The  potential  of  this 
slowly  maturing  technology  must  be  realistically  assessed  so  that  continued  R&D  funding 
can  be  appropriately  allocated  to  ensure  a  worthwhile  pay-off  in  future  years. 
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